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Abstract 



The Combined English Language skills Assessment (CELSA) was pilot 
tested in a sample of ESL classes at Golden West college during 
the Spring 1992 semester. Results indicated that CELSA scores 
are significantly related to referrals that students in all ESL 
course levels received from their instructors at the end of the 
term* Further, scores adequately differentiate among students 
who received different referral levels. Finally, the multiple 
classification scheme (HC8) was used to generate cut scores that 
comprise a placement rule recommended for implementation at GWC* 
This placement rule incorporates information from a writing 
sample to support a definitive placement for students scoring in 
the upper ranges of the distribution. The need to closely 
monitor the placement rule after implementation is stressed. 
Also, studies of possible disproportionate impact of the CELSA 
must be conducted. 



Implementation and Initial Validation of the Combined English 
Language Skills Assessment (CELSA) at Golden West College 

Steven Isonio, Ph.D. 

Introduction. 

Background. Initial placeitient into basic skills and ESL 
courses is typically accomplished with the use of standardized 
placement tests. At Golden West College, commercially available 
instruments in various approval categories have previously been 
identified and implemented for both English and mathematics. As 
of Fall 1992, only a single ESL placement test, the Combined 
English Language skills Assessment (CELSA) has met the standards 
for approval put forth by the California Community Colleges 
Chancellor's Office (California Community Colleges, 1992). In 
anticipation of the full approval of this instrument and with the 
expectation that it would eventually be implemented at Golden 
West College, the CELSA was piloted in Spring 1992. This report 
describes the pilot study, presents evidence supportive of its 
validity, and offers recommendations for the implementation of 
the CELSA at GWC. 

Description of the CELSA. The CELSA uses a cloze format, 
including parts of conversations and short dialogues. In 
developing items, common student errors offered by experienced 
language teachers were selected to be distractors in the 
multiple-choice format. This experience was combined with 
extensive item analyses to yield items that have high 
discriminating power and appropriate difficulty levels. 
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Additionally, the CELSA User Manual (Ilyin, 1992) presents some 
normative data from studies at a number of community colleges and 
adult learning programs. Results of these studies establish the 
internal consistency of the test and offer predictive validity 
evidence for its use as a placement instrument. Additionally, 
relationships between the CELSA and other measures of English 
language skills are represe ^ed, indicating a degree of 
convergent validity. 

The CELSA includes items from the beginning, intermediate, 
and advanced levels of the English Language Skills Assessment 
(ELSA) . As such, it can be used for making placement 
recommendations for the students along the full range of English 
language skill levels. Normative information and conversion 
tables with descriptive labels used by San Franciiico Community 
College ESL and Adult Education Programs are provided (see 
Appendix A) . This descriptive information can serve as an 
"anchor" for colleges seeking to implement the CELSA and to 
coordinate score ranges with their own ESL course levels. 

Method. 

Design. The basic design of the study entailed 
administering the CELSA to a sample of ESL students aud then 
relating CELSA test scores to instructor evaluations of the 
students. The evaluations were made in the form of 
referrals/recommendations for an ESL or English course the 
following semester. 



Participants. The sample consisted of a total of 377 
students enrolled in 18 sections of ESL courses during the Spring 
1992 semester. All course levels were represented (see Table 1; 
all Tables and Figures appear in the Appendix) • Analyses were 
conducted on data from students who both tested with the CELSA 
and received an instructor's recommendation (n = 349) • 

Procedure. A study designed to determine the 
appropriateness of the CELSA as a placement instrument at GWC was 
planned in early Spring of 1992. At that time, no ESL instrument 
had earned any level of approval from the Chancellor's review 
process. Further, GWC had been notified that the currently used 
instrument, the ELSA was no longer published and that its 
publisher (Newbury House) had indicated that it would not provide 
the necessary supportive information for it to be eventually 
approved. Discussions of the merits of the CELSA had taken place 
at statewide assessment workshops, and within the Coast Community 
College District at a meeting of ESL chairpersons form Golden 
West and Orange Coast Colleges, interested faculty, and 
assessment specialists. 

A number of ESL classes during the Spring 1992 semester were 
identified for inclusion in the sample. With the cooperation of 
the then Chairperson of the GWC ESL Department, instructors were 
contacted and asked for permission to use approximately one hour 
of class time during the last two weeks of the semester in order 
to administer the CELSA to students. The tests were administered 
to both day and evening classes representing the full range of 
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ESL levels. All participating instructors were further asked to 
provide copies of their rosters showing the referrals/ 
recommendations they had made for the students in their classes. 
Test data and referral information were then matched for 
analysis. 

Results. 

Overview of the Analyses. There were two major components, 
to the analysis. The first was designed to assess the 
relationship between CELSA test scores and instructor referrals. 
Specifically, correlations were computed within ESL levels and 
the median correlation was calculated across levels. These 
analyses served to establish whether the CELSA had predictive 
validity. The second set of analyses was intended to develop 
recommended cut scores (an overall placement rule) for actually 
implementing the CELSA at GWC. In this case, a variety of 
approaches was undertaken. 

Predictive Validity. The primary index of predictive 
validity of a placement test is its relationship, as evidenced by 
a correlation coefficient, to a reasonable criterion of 
performance in class; typically this criterion is the course 
grade, since ESL courses at GWC are ungraded (students receive 
"credit/no credit" instead of the letter grade scale) , the 
instructors referral/recommendation was used as the criterion 
against which to gauge the predictive ability of the CELSA. The 
referrals have an added strength in this regard; they constitute 



an instructor's judgment of the class which is most appropriate 
for the student the next semester. Like grades, they reflect an 
evaluation of work performed over the course of the term, but 
unlike grades they are more clearly a statement of the student's 
readiness for a future class. This, of course, is precisely the 
intended function of placement tests. 

Table 2 presents correlations between CELSA test scores and 
ESL instructor referral levels (considered an interval scale of 
measurement), by ESL course level. These values have been 
corrected for restriction of range due to the use of intact, pre- 
selected groups (Matriculation Local Research options Committee, 
1991) • The r values range from .41 (for ESL 004) to .79 (for 
both ESL 001 and the combined ESL 900/001) , with a median value 
of r = .59. Collapsing across levels, the overall relationship 
between CELSA scores and instructor referrals is somewhat high 
(£ = .72). In all cases, the correlation coefficients are 
statistically significant. Thus, there is clear evidence for the 
predictive validity of the CELSA as a placement tool for ESL 
courses at GWC. 

Development of a Placement Rule, since the instructor 
z^eferrals indicate which course is considered most appropriate 
for a student the following semester, they were used to create 
sub-distributions for comparison purposes in support of the 
development of cut scores. Table 3 shows selected descriptive 
statistics for the sub-distributions of students receiving each 
of the seven referral levels. As can be seen, the CELSA is able 
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to adequately differentiate among referral levels, as evidenced 
by the spread of mean and median score values across levels 
(Figure 1 depicts mean CELSA scores, by referral level) . An 
analysis of variance performed on CELSA scores by referral levels 
indicated highly significant differences among the levels [F (6, 
342) =91.21, E < .001.]. The Tukey-Kramer test was used for 
multiple comparisons because of its appropriateness in cases 
where there is at least a moderate imbalance in sample sizes 
(Dunnett, 1980). This analysis revealed that all pairwise 
difference comparisons are statistically significant, except for 
the difference between the values for (ESL 001) and (ESL 002) ; 
these results are summarized in Table 4. Finally, the 
coefficients of skewness indicate that*, with the exception of 
referral level 2, the sub-distributions are relatively 
symmetrical . 

The CELSA User Manual (Ilyin, 1992) describes and recommends 
the Multiple Classification Scheme (MCS; developed by Cooley and 
Lohnes, 1971) . It incorporates mean and standard deviation 
values for adjacent groups in identifying a score level (a cut 
score) that best differentiates the two groups. The formula for 
using the MCS is presented in Appendix A of this report. The 
result of the MCS computations and resulting recommended cut 
scores are presented in Table 5. The second column of that Table 
contains the MCS-calculated cut scores which should serve as the 
tiPPgr limits of the score ranges. These values are rounded 
upward and used to define placement score ranges depicted in the 



last column of Table 5, As can be seen, the score ranges for the 
lowest group is "0 to 23". Ten sr^ore values (range 24 to 33) are 
associated with ESL 001; the next 8 and then 9 score values are 
associated with ESL 002 (34 to 41) and ESL 003 (42 to 50) , 
respectively. The next five score values (51 to 55) would 
reconunend placement into ESL 004 and the following five (56 to 
60) to ESL 005. Finally, scores of 61 or greater would result in 
placement recommendations to English 10, 

As shown in Table 5, the score ranges differentiating the 
highest placements are not as wide as those associated with the 
lower levels. An ESL placement writing sample (PWS) has been 
used at Golden West College to assist in making definitive 
placements for students scoring in the upper ranges of score 
values^ Jiolistic evaluations of the ESL PWS have previously been 
shown to be reliable (IsoniO; 1992). The ESL PWS can be 
incorporated into the model developed for the CELSA in a similar 
way so as to make definitive placements. A reasonable scheme for 
doing so would be to evaluate writing samples for all students 
scoring in the ranges associated with ESL 005 and English 10. In 
this way, those students scoring 56 to 60 would either retain the 
ESL 005 recommendation or receive a placement recommendation to 
ESL 004 or to English 10. Similarly, students scoring 61 or 
above would be referred to ESL 005, or to English 10, depending 
upon the evaluation of their writing sample. (The placement 
scheme that incorporates the ESL placement writing sample is 
depicted in Table 6.) In short, both the multiple 



classification scheme and the examination of CEL£\ score sub- 
distributions, in a variation of the contrasting groups approach, 
support the appropriateness of the recommended initial placement 
rule* 

Sximmary and Discussion* 

Evidence has been presented that supports the value of the 
CELSA as a placement instrument at Golden West College, Using 
instructor referrals/recommendations as the primary criterion 
against which to gauge CELSA scores, impressive predictive 
validity coefficients were obtained across the range of ESL 
course levels thereby establishing that there is a significant 
relationship between CELSA scores and this criterion. Further, 
both analyses using the MCS approach and the contrasting groups 
method to develop a rule for generating course placement 
recommendations were performed. These approaches converge in 
their support of a placement rule for ESL levels 900 through ESL 
5, and above. Further, to clarify the placement associated with 
high score ranges, the ESL Placement Writing Sample should be 
used to make a definitive placement for students scoring in the 
ranges associated with ESL 005 and English 10/100. 
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Recommendations • 



1. The Combined English Language Skills Assessment (CELSA) 
should be adopted and implemented as the primary ESL 
placement measure at Golden West College. This 
instrument has received provisional approval by the 
Chancellor's Office and the evidence presented in this 
report supports its appropriateness as a placement tool 
at GWC. 

2. The placement rule depicted in Table 6 should be used 
initially. It was derived from the Multiple 
Classification Scheme recommended by the author of the 
CELSA and is consistent with an examination of the 
graphical representation of CELSA score sub-distributions 
for contrasting groups* 

3. This placement rule should be monitored for its 
effectiveness. Instructors and students in a sample of 
ESL courses should provide midterm ratings of the 
appropriateness of the placement. These ratings would 
therefore serve to corroborate the initial placement 
scheme or provide a basis for adjustments deemed 
necessary. 

4. Information from other measures should be collected and 
analyzed to support the eventual development of a 
multiple measures placement model for ESL courses. 

5. Studies of possible disproportionate impact and 
differential validity of the CELSA as a placement 
instrument at Golden West College should be conducted. 

6* Item analyses by referral level subgroups should be 

performed to support the ongoing examination of the ESL 
curriculum. 
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APPENDIX B-1 

ADULT EDUCATION E8L PROFZCIEMCY LEVELS 

The San Francisco Cororounity College centers' revised ESL Master 
Plan describes eight levels of instruction beginning with level 50, 
which is a basic literacy course as well as an introductory course for 
students with zero proficiency in speaking, reading, and comprehendinq 
English. For obvious reasons, formal testing is not done at this 
level. Levels 100-700 are described below. These levels lend 
themselves well to the two CELSA test forms. Conversion tables for 
placement are presented in Tables 1 and 2 and on the keys. 

LQWEE PEGINNING (ESL 100) i Designed for the student with little 
knowledge of English, the basic goal is for students to begin to 
communicate and conduct the affairs of daily life in spoken and written 
English. High frequency vocabulary is eniphasized and regular forms of 
the present, past, and future are covered in the standard 
transformations (affirmative, negative yes/no and WH-questions) . 

CPEEE BEGINNING (gSL 200) ; Students continue to gain simple 
communicative ability to function in practical areas necessary for 
survival m the United States. Although students read short authentic 
passages and simple conversations, emphasis continues to be on oral 
production and listening comprehension with an expansion of the 
structures of level 100 to include irregular past forms, present 
perfect, modals, and comparison using adjectives and adverbs. 

INTERMEDIATE lESL ?00) ; A period of transition when students 
assimilate previously learned material and begin to develop fluency as 
well as accuracy both in spoken and written English. Students read both 
short authentic materials as well as adapted riading selections! 
Complex sentences using noun, adjective and adverb clauses are 
intr9duced; past forms in the continuous and perfect tenses and 
conditionals in the present and future are stressed. ^^"^^^ ^""^ 

?PPgR INTERMEDIATE fFST, 4P0) ; Advanced grammatical structures such as 
complex sentences using embedding and relative clauses; passive fo^-rj 
future perfect and continuous, unreal conditionals, and Complex word 
order are introduced. Students continue to read authentic matlJiSls and 
selections appearing in ESL texts. Students often are in vocational 
courses for training or upgrading, vocational 

I^WEP ADVANCED fFSL 5QQ) ; Students learn -to use more advanced 
Se?f2J?^nn J'?^^"^^"^ H^odal auxiliaries in past forSs? ?u^2?e perfect 
perfect continuous forms, and perfect conditional forms The? aiSo 

llelicln Ena??^i"^^'S^-^r°P"^ "^^"^ intonatiSn; Ind'lAythm^f "spoken 
|Sp?Srf" iSe.cSSJ^rSlSSS^SfiS^^SI SS^ 

^PEES APVANgEp i£SL $Q0) ; Students review materials from orevious 
courses and prepare for high school diploma courses college e^?r«nno 
or more advanced vocational training. A thorouah ?4view of PnSH^h ' 

lltZllL "^ding sJ^ills, cLpoJitio^? iSd'^I^iden? reslarch 

projects are basic components of the course, A division 

awarded upon successful completion of thirievel? certificate is 

fB^5EDJBnJS_lML.2I^ students work on the skills necessarv to 
function m academic situations in non-ESL classes on hiSh IShLl 
SSiiw??' ^""^ university levels. Emphasis is on the expansion o? ' 
s?i??S^^''X'^?""^"*''i^??^ analvsis and comprehension an5^|?he? readina 
•SpiisizeS ^'""^ including composition and research arfals? 
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Multiple Classification Scheroei 
to Establish Cut Scores Between Adjacent Levels 



LEVEL 1 
LEVEL 2 

CUTOFF = ^i(S2) ^ X2(Si) 

Si + S2 

« 

• where S = Standard Deviation 
X = Mean 

Subscripts 1 and 2 indicate level 1 and 2, respectively 



1 (SEE: Cooley & Lohnes, 1971) 
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Table 1 

Number of Students rand Sections > Sampled in the CELSA Pilot 
Study. Bv ESL Course Level 



Course Level 


Number of Sections 


Number of Students 


ESL 900 


1 


24 




2 


45 


ESL 001 


1 


27 


ESL 002 


3 


67 


ESL 003 


2 


57 


ESL 004 


3 


87 


ESL 005 


3 


70 


Total 


15 


377 
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Table 2 

Predictive Validity Coefficients, Bv Course Level 



Course Level Validity Coefficient 



ESL 


900 


. 52 


ESL 


900/001 


.79 


ESL 


001 


.79 


ESL 


002 


.70 


ESL 


003 


. 59 


ESL 


004 


.41 


ESL 


005 


.47 



Median r = . 59 



Note: ail validity coefficients for individual course levels 
have been corrected for restriction of range. 
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Tabl« 3 

Selected CELSA Sc ore Descriptive Statistics of Referral Level 
Sub-Distributions 



CELSA Score 



Referral Level 


Mean 


Median 


St . Deviation 


ESL 900 


16. 50 


15. 5 


12 . 19 


ESL 001 


30.72 


30 


11.95 


ESL 002 


33 .72 


37 


9. 90 


ESL 003 


46.74 


47 


8.93 


• ESL 004 


52.97 


54 


9. 67 


ESL 005 


56.58 


57 


8.86 


ENG 010 


62.23 


63 


9.40 


Total 


48.32 




14 .38 
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Table 4 

Results of Multiple Comparison Tests For Differences 
Between ^ ^irs of Means 



Levels / Means In Critical Obtained 
the Comparison Difference fp<, 05) Difference 



ESL 900 
ESL 001 


(16.50) 
(30.72) 


8.60 


14.22* 


ESL 001 

e<;l 002 


(30.72) 
^ ^ <j . 1 ^ ) 




Z • 7 o 


ESL 002 
ESL 003 


(33.70) 
(46.74) 


3.31 


13. 04* 


ESL 003 
ESL 004 


(46.74) 
(52.97; 


3.20 


6.23* 


ESL 004 
ESL 005 


(52.97) 
(56. 58) 


3.23 


3.61* 


ESL 005 
ENG 010 


(56. 5S) 
(62.23) 


3.59 


5.65* 


* p < . 


05 






(Note: 


Comparisons were 


based on 


the Tukey-Kramer procedure.) 



Table 5 

Multipl e Classification Scheme (MCS) Placement Rule 



Referral Level 


MCS Cut (Upper) 


Placement 
Score Range 


ESL 900 


22.76 


<= 23 


ESL 001 


32.42 


24-33 


ESL 002 


40.25 


34-41 


E§L 003 


. 49.73 


42-50 


ESL 004 


54.85 


51-55 


ESL 005 


59.32 


56-60 


• ENG 010 




>=61 



Table 6 

Recoinniended Initi al Placement Rule for CELSA at 
Golden W est College ' 



Placement Recommendation 

ESL 900 
ESL 001 
ESL 002 
ESL 003 
ESL 004 
. ESL 005 or higher* 
English 010 or higher* 



CELSA Score 
Range 

<= 2 3 
24 - 33 
34 - 41 
42 - 50 
51 - 55 
56 - 60 
>= 61 



*ESL Placement Writing Samples are evaluated for students scorinq 
rn these ranges. 
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